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An apparatus and method for dynamically varying the frame 
rate of an image sequence is disclosed. In one embodiment, the 
mage sequence ,s encoded and stored at different frame rates 
le.g., 30, 25, 20 fps and so on). Alternatively, only the motion 

stored ' e ' 8 " m0ti ° n VeCt0rS • f ° r ° thcr fra ™ ra ^ are 



START 



308 



TRANSFER IMAGE 8BOU&0CB TO 




CLIENT AT A OBTNED FRAME RATE 




31Q 






^^CHAHGE^^ NO 






<^FRAME RATE ? ^> » 








Tyes 





, IMAGE ^ . 

Sequence storeo\* es 
i new frame rate' 



RETRIEVE 
ENCOOED IMAGE 
SEQUENCE AT NEW 
FRAME RATE 
34Q 



ENCODE ORIGINAL (MAGE SEQUBMC6 
AT NEW FRAME RATE USING STORED 
MOTION INFORMATION 

ass 




gSDOCID: <WO_0001151A1_|. 



r 



WO 00/01151 



PCT/US99/14510 



APPARATUS AND METHOD FOR DYNAMICALLY CONTROLLING 
THE FRAME RATE OF VIDEO STREAMS 

This application claims the benefit of U.S. Provisional Application 
5 No. 60/090,814 filed on June 26, 1998, which is herein incorporated by 
reference. 

The invention relates generally to the field of digital multimedia 
communications over a network, e.g., computer networks. More specifically, 
0 the invention relates to a method of storing motion information for a 
sequence of images in different frame rates to effect dynamic frame rate 
control over a network. 



BACKGRO UND OF THE DISCLOSTTRF. 
15 The increasing demand for digital video/audio information presents 

an ever increasing problem of transmitting and/or storing an enormous 
amount of information. For example, the growth of the Internet and the 
proliferation of Internet browsers in recent years has enabled millions of 
users to access various information in a matter of seconds. Most of the 
20 information is currently in the form of text and/or still images and graphics. 
Generally, this type of information can be quickly downloaded and 
displayed by a user. More recently, audio and video clips or sequences have 
been added to numerous webs sites. Without an increase in bandwidth, 
accessing these clips requires more computational cycles and access time 
25 due to the sheer size of the image sequences. 

Two methods of accessing an image sequence across a network are 
available. First, the user can download the compressed bitstream in its 
entirety, and then decode and display the image sequence locally using the 
user's computer. One drawback of this method is the length of time that a 
30 user must wait while the image sequence is being downloaded. Another 
drawback is that the user may need to reserve a large storage space to 
accommodate a large image sequence. 
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at different frame rates (e.g., 30, 15, 10 and 5 fps). This embodiment 
increases the storage requirement for a server. 

In a second embodiment, only the motion information, e.g., motion 
vectors, for the other frame rates are stored. Namely, the image sequence is 
5 first encoded and stored at a preferred frame rate and then the same image 
sequence is encoded at one or more different frame rates. The encoding 
process at other frame rates can be performed off-line. However, only the 
motion information is saved and stored, e.g., in "motion files", for the other 
frame rates, thereby reducing the storage requirement of the server. Thus, 
10 when it is necessary to re-encode the image sequence to accommodate a 
different frame rate, motion vectors are obtained from a storage device, 
instead of being computed. This method enables the image sequence to be 
re-encoded at a different frame-rate in real-time. Additionally, the motion 
files are significantly smaller than the corresponding compressed image 
15 sequence. 



BRIEF r)Rqr!T?TPTT Q N OF TTTK DRAWINGS 
The teachings of the present invention can be readily understood by 
considering the following detailed description in conjunction with the 
20 accompanying drawings, in which: 

FIG. 1 illustrates a block diagram of a network environment having a 
server incorporating the dynamic frame rate control of the present 
invention; 

FIG. 2 illustrates a block diagram of an encoder incorporating the 
25 dynamic frame rate control of the present invention; 

FIG. 3 illustrates a flowchart of a method for effecting the dynamic 
frame rate control; 

FIG. 4 illustrates a block diagram of a motion file of the present 
invention; and 



30 



FIG. 5 illustrates a block diagram of an encoding system of the 
present invention. 
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server 110 having an encoder 112, a decoder 114 and a storage 116. The 
encoder 112 is an encoder capable of performing motion estimation and 
retrieving motion information. As such, the encoder 112 can be 
implemented as an MPEG or H.263 compliant encoder. 
5 In operation, one or more image sequences are pre-encoded and 

stored in storage 116 at a particular bit rate and frame rate. If a client 130 
requests an image sequence from the server 110 that matches the bit rate 
and frame rate of the pre-recorded image sequence, then the server will 
simply forward the image sequence directly to the client via the network 
10 without further processing. 

If the client requests an image sequence at a different frame rate 
from the frame rate in which the image sequence was encoded, then the 
server 110 must decode the encoded image sequence using the decoder 114. 
It should be noted that if the original image sequence is also stored in the 
15 server 110, then the decoding operation and the decoder 114 itself can be 
omitted all together, since the original image sequence is readily available. 

Once the original image sequence is obtained (either through 
decoding the encoded image sequence or retrieving the original image 
sequence from storage), the encoder 112 can now re-encode the image 
20 sequence at the new frame rate. However, the encoder 112 does not have to 
compute the motion information from the image sequence. Instead, the 
encoder 112 reads the motion information from a "motion file" associated 
with the new frame rate, e.g., 25 fps, 20 fps, 15 fps, 10 fps, 5 fps and so on. 
Since the majority of the computational cycles in an encoding process is 
25 spent in generating motion information, e.g., motion vectors, encoder 112 
can now quickly re-encode the image sequence in real-time using the stored 
motion information to accommodate a frame rate as requested by a client. 

Effectively, a second knob is provided to a client for varying the frame 
rate of the image sequence. By reducing the frame-rate of the image 
30 sequence, the bandwidth will be reduced without further loss in the spatial 
quality (temporal quality would be reduced instead). The present dynamic 
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of information that is transmitted on a channel because only the changes 
between the current and reference frames are coded and transmitted. 

Alternatively, the present invention incorporates a storage 116 for 
storing motion files for an image sequence at various frame rates. Namely, 
5 if the current image sequence on path 210 has been previously encoded and 
the motion information for the image sequence is stored in storage 116, then 
the motion estimation process performed by the motion estimation module 
240 is bypassed and the associated motion vectors for a particular frame 
rate are simply read from a motion file stored in storage 116. This process 
10 greatly increases the speed with which the image sequence is encoded. 
However, if the image sequence was not previously encoded or the motion 
information for a desired frame rate was not previously generated, then the 
motion vectors are generated using motion estimation module 240. The 
resulting motion vectors can then be stored in a new motion file in storage 
15 116. 

The motion vectors from the motion estimation module 240 or the 
storage 116 are received by the motion compensation module 250 for 
improving the efficiency of the prediction of sample values. Motion 
compensation involves a prediction that uses motion vectors to provide 
20 offsets into the past and/or future reference frames containing previously 
decoded sample values that are used to form the prediction error. 

Furthermore, prior to performing motion compensation prediction for 
a given macroblock, a coding mode must be selected. In the area of coding 
mode decision, MPEG provides a-plurality of different macroblock coding 
25 modes. Specifically, MPEG-2 provides macroblock coding modes which 
include intra mode, no motion compensation mode (No MC), 
frame/field/dual-prime motion compensation inter mode, 
forward/baclcward/average inter mode and field/frame DOT mode. 

Once a coding mode is selected, motion compensation module 250 
30 generates a motion compensated prediction (predicted image) on path 252 of 
the contents of the block based on past and/or future reference pictures. 
This motion compensated prediction on path 252 is subtracted via 
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back to the prediction signal from the motion compensation module via 
summer 255 to produce a decoded reference picture (reconstructed image). 

FIG. 3 illustrates a flowchart of a method 300 for effecting the 
dynamic frame rate control of the present invention. More specifically, FIG. 
5 3 illustrates a method for allowing a client to dynamically change the frame 
rate of an image sequence. 

Method 300 starts in step 305 and proceeds to step 310, where an 
image sequence is transferred to a client at a predefined frame rate. It 
should be noted that if the frame rate requested by the client does not 
10 match the frame rate of the pre-recorded image sequence, then step 310 is 
bypassed and method 300 proceeds to step 320. 

In step 320, method 300 queries whether the frame rate is changed. 
The frame rate of an image sequence can be changed for a number of 
different reasons, e.g., as requested by the client, or the bandwidth of the 
15 communication channel has changed and is detected by the server. If the 
query is negatively answered, then method 310 returns to step 310 and 
continues sending the image sequence to the client at a predefined frame 
rate. If the query is affirmatively answered, then method 300 proceeds to 
step 330. 

20 In step 330, method 300 queries whether the image sequence is 

stored in the desired new frame rate. Namely, method 300 is inquiring 
whether there is a stored image sequence that was previously encoded at 
the desired frame rate. 

In one embodiment of the present invention, each image sequence is 
25 encoded at a plurahty of different frame rates (stored information). For 
example, an original image sequence can be encoded in three different 
frame rates, such as 30 fps, 25 fps and 20 fps. If the frame rate is changed 
from 30 fps to 25 fps, the server simply reads the image sequence from a 
different file. However, this embodiment dictates a large storage 
30 requirement for the server. 

Returning to step 330, if the query is affirmatively answered, then 
method 300 proceeds to step 340, where the stored encoded image sequence 
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More specifically, the presence of motion information field 420 is 
implemented as a single bit per macroblock. A zero ("0") indicates that 
there are no motion vectors associated with a current MB (which would be 
the case if the MB is not encoded (COD = 1) or if it is encoded in INTRA 
5 mode). A one (T) indicates that there is one motion vector for the current 
macroblock, where the motion vector is stored in the motion information 
field 440. 

Optionally, the mode information field 430 is provided to further 
distinguish between the COD = 1 case, and the INTRA mode. Namely, if 

10 the encoder is informed that no motion information is available for a current 
block, then the encoder must determine whether the absence of motion 
information is due to the lack of motion in the current block or the 
macroblock is being Intra coded. Generally, an encoder will perform a Sum 
of Absolute Difference (SAD) calculation when no motion vector is available 

15 for the purpose of determining whether the MB should be INTRA coded. 
Namely, the absolute difference between each corresponding pixel value, 
e.g., within an area such as a macroblock between successive frames is 
performed. Next, a sum of the all the absolute difference values is 
compared to a predicted SAD, which serves as a threshold to measure the 

20 degree of change in the macroblock. The Mode Decision is then selected 
accordingly, i.e., Intra coding for substantial change and Inter coding for 
insubstantial change. 

Unfortunately, these SAD calculations are time consuming and will affect 
the performance of the real-time encoder. 

25 As such, the mode information field 430 provides an additional bit 

such that the combination of the presence of motion information field and 
the mode information field allows the implementation of a variable length 
code (one or two bits) in the motion file that can be used to indicate both the 
COD and the MB type (Intra or no motion). For example, a '0' would 

30 indicate COD = 1, a '10' would indicate mode = INTRA and a '11' would 
indicate mode = INTER and presence of a motion vector. The 
implementation of the reason for absence of motion information field 430 
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computer 510. The encoder 516 and the decoder 518 can be physical devices 
which are coupled to the CPU 512 through a communication channel. 
Alternatively, the encoder 516 and the decoder 518 can be represented by a 
software application (or a combination of software and hardware, e.g., 
5 application specific integrated circuits (ASIC)) which is loaded from a 

storage device and resides in the memory 512 of the computer. As such, the 
encoder 200 of the present invention can be stored on a computer readable 
medium, e.g., a memory or storage device. In turn, the motion information, 
i.e., motion files and the encoded image sequence at various frame rates, 
10 which are generated by the encoder 516, can also be stored on a computer 
readable medium, e.g., RAM memory, magnetic or optical drive or diskette 
and the like. 

The computer 510 can be coupled to a plurality of input and output 
devices 520, such as a keyboard, a mouse, a camera, a camcorder, a video 
15 monitor, any number of imaging devices or storage devices, including but 
not limited to, a tape drive, a floppy drive, a hard disk drive or a compact 
disk drive. The input devices serve to provide inputs to the computer for 
producing the encoded video bitstreams or to receive the sequence of video 
images from a storage device or an imaging device. 
20 Although various embodiments which incorporate the teachings of 

the present invention have been shown and described in detail herein, those 
skilled in the art can readily devise many other varied embodiments that 
still incorporate these teachings. 
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6. A data structure stored on a computer readable medium comprising: 
a start of frame code field; and 
motion information field. 

5 7. The data structure of claim 6, further comprising a presence of 
motion information field. 



8. The data structure of claim 7, further comprising a mode information 
field. 



9. A server for dynamically controlling the frame rate of an image 
sequence, said server comprising: 

a means for transferring the image sequence at a first frame rate; 

a means for receiving a request for a change from said first frame 
15 rate to a second frame rate; and 

wherein said transferring means transfers the image sequence at said 
second frame rate by retrieving stored information of said image sequence 
in accordance with said second frame rate. 

20 10. The server of claim 9, wherein said stored information is a stored 
motion information of said image sequence encoded at said second frame 
rate. 
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